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This patent application claims priority under 35 U.S.C §1 19 of provisional patent 
application, Serial No.: 60/178,228, filed January 26, 2000, which is hereby incorporated by 
reference as if set forth in its entirety herein (including the source code). 

Source Code Listings 

This application is filed with an appendix containing source-code listings. 

Background of the Invention 

1. Field of the Invention 

The present invention relates to interactive computer systems, and in particular to 
a method and apparatus for facilitating wireless, full-body interaction between a human 
participant and a computer-generated graphical environment including textual data. 

2. Description of Related Art 

So-called "virtual reality" ("VR") systems enable users to experience computer- 
generated environments instead of merely interacting with them over a display screen. Such 
systems typically require the user to don goggles, through which he or she perceives the virtual 
environment, as well as sensors that encode the user's gestures as electrical signals. The user 
reacts naturally to the changing virtual environment, generating signals that the computer 
interprets to determine the state and progress of the presented environment. 

In order to encode a sufficiently broad spectrum of gestures to facilitate natural 
interaction, VR systems ordinarily require the user to wear, in addition to the goggles, at least one 
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"data glove" to detect hand and finger movements, and possibly a helmet to detect head 
movements. Full-body systems, which encode movements from numerous anatomical sites to 
develop a complete computational representation of the user's overall body action, require many 
more sensors; however, such systems, would be capable of projecting the user fully into the 
5 virtual environment, providing the user with greater control and a heightened sense of 
participation ideally suited to interactive simulations. 

Unfortunately, numerous practical difficulties limit the capacity of current VR 
systems to achieve this goal. The nature of the interaction currently offered, even with full-body 
sensor arrays, is limited. The computational demands placed on a system receiving signals from 

3|) many sensors can easily overwhelm even large computers, resulting in erratic "jumps" in the 
visual presentation that reflect processing delays. Moreover, no matter how many sensors 

H surround the user, they cannot "see" the user, and therefore cannot integrate the user's true visual 

)L image into the virtual environment. 

h jj Economic and convenience factors also limit sensor-type VR systems. As the 

35 capabilities of VR systems increase, so do the cost, awkwardness and inconvenience of the 

sensor array. The sensors add weight and heft, impeding the very motions they are intended to 
detect. They must also ordinarily be connected, by means of wires, directly to the computer, 
further limiting the user's movement and complicating equipment arrangements. 

In order to overcome the limitations associated with sensor-based VR systems, 
20 researchers have devised techniques to introduce the user's recorded image into a virtual 

environment. The resulting composite image is projected in a manner so that it may be viewed 
by the user, enabling the user to observe his or her appearance in and interaction with the virtual 
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environment. 



Three such approaches include the VideoPlace system (see, e.g., M. Krueger, 



Artificial Reality II (1991) and U.S. Pat. No. 4,843,568), the Mandala system (see, e.g., Mandala 
VRNews, Fall/Winter 1993; Vincent, "Mandala: Virtual Village" and Stanfel, "Mandala: Virtual 
Cities," Proceedings of ACM SIGGRAPH 1993 at 207-208 (1993)), and MIT's system disclosed 
in U.S. Pat. No. 5,563,988. Unfortunately, these systems exhibit various limitations. For 
example, Krueger's VideoPlace requires a special background and ultraviolet lamps, and extracts 
and represents only the user's silhouette. The Mandala system can integrate the user's full image 
within the virtual environment it creates, but requires a chroma-key blue background. The 
system developed by MIT provides a 3-Dimensional spatial video interaction with a user and is 
directed to detecting and analyzing gestural information of the user to cause predetermined 
responses from graphically generated objects within the virtual environment (e.g., if a virtual user 
pets a graphically generated dog within the virtual environment, the dog may wag its tail). 
Regardless of the systems, all of the above-described VR systems employ complex video analysis 
and tracking techniques to detect and follow the boundaries of the user's image within the virtual 



environment. 



The disclosure of US Patents Nos. 5,563,988 of Maes et al, and 4,843,568 of 



Krueger et al. are hereby incorporated by reference as if set forth in their entirety herein. 

Applicant has recognized the need for a simple 2-dimensional video-generated 
real-time interaction system wherein text data is easily introduced into a virtual environment, as 
viewed on a monitor or on a projection screen. The inputted text data is manipulated in the 
virtual environment by video representation of a human participant (a "virtual user"). Through 
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movements in the real world, the human participant or virtual user freely interacts with the 
inputted text data in the virtual environment as the human participant (and other people watching 
the monitor or projection screen) absorbs the teachings of the text data as he or she interacts with 
the text data in the virtual environment. 

Objects of the Invention 

It is an object of the present invention to provide a wireless VR system that easily 
accepts text data from a text input to be displayed onto a monitor (or projection screen) so that 
the font, size, shape, and/or movements of words and/or characters (e.g, letters, symbols and 
numbers) of the inputted text data may be altered by a virtual user (a real-time video 

representation of a human participant). 

It is another object of the invention to enable the virtual user to interact with 
computer-generated, visually represented text data in such a manner that allows the inputted text 
to be readable within the virtual environment so that the virtual user may learn, understand, or 
otherwise experience the meaning of the textual data as he or she interacts with the text within 

the virtual environment. 

It is another object of the invention to provide an interaction display system 
wherein text data and the video image of a user is combined within a virtual environment to 
promote marketing, advertisement, sales, and entertainment. 

It is another object of the invention to provide an interaction display system 
wherein text data and the video image of a user is combined within a virtual environment and 
wherein the text data viewed within the virtual environment may be changed in realtime in 
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response to actions by the user or by others. 

It is another object of the invention to provide an interaction display system 
wherein text data and the video image of a user is combined within a virtual environment and 
wherein the text data is imported from a local text file. 

It is another object of the invention to provide an interaction display system 
wherein text data and the video image of a user is combined within a virtual environment and 
wherein the text data is imported from a remote site over the Internet. 

It is another object of the invention to provide an interaction display system 
wherein text data and the video image of a user is combined within a virtual environment and 
wherein the text data is inputted directed from a keyboard. 

The foregoing and other objects of this invention can be appreciated from the 
Summary of the Invention, the drawing Figures and Detailed Description. 



Summary of the Invention 

In accordance with the invention, a controlling software program instructs an 
interactive display system. The interactive display system includes an electronic camera (e.g., 
charge-coupled device or "CCD") which intermittently records (in realtime) a human 
participant's (a user) image and inputs the video image into a computer. The computer creates a 
digitized version of the inputted image (e.g., a bitmap of pixels) and thereby creates a "virtual 
user image". According to one embodiment, the camera is positioned behind a large viewing 
screen which faces the user and records the user's image through an opening provided within the 
screen. The computer further receives a data record including text data and text controlling 
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parameters, preferably including at least a fall rate value. The data record may be supplied to the 
computer using any of several conventional memory mediums, such as a hard drive memory, disc 
memory, compact disc, or the data record may only include a text string which is supplied to the 
computer directly from the Internet through a connected modem, for example, or through the use 
of a connected keyboard. Alternately, the text may be inputted to the computer directly from a 
word processing program, such as Corel's WordPerfect®, or Microsoft Word® as a text file or 
by "copying" selected text from an open text file directly into the present software program. 

According to the invention, upon receiving a captured video image of the user, the 
software program (see appendix for source code) first parses the text data into lines of text. The 
program determines which line or lines of text should be displayed on the screen and further 
determines the destination of each character within any line that is to be displayed on the screen, 
according to the fall rate assigned to each character. The program then compares the pixel color 
value of "destination pixels" (pixels located at the destination of each particular character) with a 
threshold color value. If the color value of the destination pixels is below the threshold value 
(i.e., lighter), then the particular character will be positioned at a destination according to its 
prescribed fall rate. If, however, the color value of the destination pixels exceeds the threshold 
value (i.e., darker), then it is determined that the virtual user's image is located at the destination 
of the particular character of the line of text. In such instance, the program locates a suitable new 
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destination positioned vertically above the original destination which is lighter in color value 
than the threshold value. 

Once all the characters of all the lines of text intended to be projected onto the 
screen are provided with suitable destinations, the program instructs the computer to display all 
of the characters on the screen at their respective destinations with the particular captured video 
frame. The process continues when a new image frame is available from the camera. 

The program analyzes each captured image as it acquires at a prescribed sampling 
rate. The sampling rate is dependent on the speed of the video card components and drivers, the 
speed of the computer, and light sensitivity of the digital camera. 

According to the invention, as the individual characters of a single line make their 
way down along the projection screen, the program keeps track of when a predetermined number 
of characters (e.g., at least half of them) travels a predetermined distance from the top of the 
viewing screen (such as the halfway point). If this is the case, the program causes the entire line 
of text to fade at a prescribed fade rate. When one line fades at the lower portion of the screen, 
another line appears at the top of the screen, with its characters beginning to fall It is preferred 
that at least three lines of text be displayed on the screen at any given time. 

Another aspect of the invention displays each line of text data as a separate color 
so that viewers may easily discern the letters or characters from different lines of inputted text, 
allowing the lines of text to be read relatively easily, as the letters fall towards and rest on various 
darker elements located within the virtual environment, such as the arm of a virtual user and 
perhaps a "virtual umbrella image" carried by the virtual user. 
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According to another aspect of the invention, each character of an inputted text 
string is horizontally held so that each character is restricted to vertically movement as it 
descends on the viewing screen at its prescribed fall rate or ascends upwardly with similar 
upward movement by the virtual user. 
5 According to another embodiment of the invention, the program generates a 

shadow zone of dark pixels which are projected onto the viewing screen. The shadow zone is 
aligned with the optical input of the digital camera so that the camera will be protected from the 
relatively bright light of the projector. The computer generated shadow zone helps prevent the 
images recorded by the camera from becoming "washed out" by the light from the projector 
IW shining on the camera's lens. 

M Brief Description of the Drawings 

* The foregoing discussion will be understood more readily from the following 

~* detailed description of the invention, when taken in conjunction with the accompanying 

A5 drawings, in which: 

Fig. 1 is a schematic illustrating an interactive display system including a 
projection screen, a pickup camera, a projector, a computer, and a text input, according to the 
invention; 

Fig. 2 is a perspective view of the interactive display system of Fig. 1, showing 
20 the field of view of the camera, according to the invention; 

Fig. 3 is block diagram of the interactive display system, according to the 

invention; 
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Fig. 4 is a illustration of an exemplary text string to be inputted into the interactive 
display system, according to the invention; 

Fig. 5 is a illustration of an exemplary data record which includes the text string 
of Fig. 4 and further includes text-controlling parameters, according to the invention; 

Fig. 6 is a flow diagram showing steps of obtaining and combining video and text 

data, according to the invention; 

Fig. 7 is a front view of the viewing screen, illustrating an exemplary displayed 
virtual image including a virtual user holding a virtual umbrella, and further including falling 

text, according to the invention; 

Fig. 8 is an enlarged view of a portion of the virtual umbrella of Fig. 7 and select 

letters of the text, according to the invention; and 

Fig. 9 is a enlarged partial front view of the viewing screen showing details of a 
central opening and a computer generated shadow zone, according to the invention. 

Detailed Description of the Preferred Embodiments 

Referring to Figs. 1, 2, and 3 the interactive display system 10, according to the 
present invention, is shown including a large display screen 12 having a generally central and 
small opening 14, an image projector 16 positioned to project an image (along projection beam 
17) onto the front surface of screen 12, and a camera 18, preferably a mini black and white 
CMOS-type video camera (alternately a Connectrix QuickCam® by Logitech® with appropriate 
platform drivers may be used). The camera 18 is positioned behind screen 12 with its image- 
receiving lens aligned with opening 14, so that a person (a user) standing in front of screen 12 (in 
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the area represented by reference numeral 15, in Figs. 1 and 2) and within the field of view 19 of 
the camera 18, is recorded by the camera. The camera 18 is preferably sufficiently light 
sensitive to "capture" repetitive images of the user in apparent realtime (i.e., generating a 
sufficient number of consecutive images so that when the sampled images are eventually 
projected, the motions of the user appear smooth and continuous). It is preferred, although not 
shown, that a rear surface or wall be located opposite the screen 12, behind the user, and that this 
rear surface be white and illuminated. Although the rear surface is not necessary for the present 
system to operate, it does allow the present program to quickly and easily distinguish the virtual 

user from the background. 

The present interactive display system 10 further includes a computer 20 having a 
memory 21 and a text input 22, as shown in Fig. 1 . Computer 20 may be conventional, and 
preferably includes a fast central processing unit (CPU), such as Intel's Pentium II® or higher. 
The video card of the computer preferably includes enough VRAM to display 640 X 480 
resolution at high color (16 bit). If a CMOS type camera 18 is being used behind the large screen 
12, then the computer 20 further requires a video capture card 24 to create a bitmap of the analog 
image of the CMOS camera 18.. If a Connectix®-type camera is used (or equivalent), no capture 
card 24 is necessary because the Connectix®-type camera uses software to digitize a recorded 
image. The computer 20 preferably uses Microsoft's Windows 98® or Microsoft's Windows 

NT® operating systems. 

As is understood by those skilled in the art, the video capture card 24 is used to 
receive analog image data from the camera 1 8 and digitize the image by generating a bitmap (the 
present program can control the rate of sampling). During each sampling, as described below, 
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the video card 24 captures a frame of image data from camera 1 8 as the camera records the user 
and digitizes the frame into array of picture elements, or "pixels," called a bitmap. Depending on 
the format, each pixel within a bitmap is identified by an RGB color code (a three number code 
which identifies the exact color of the pixel). The bitmap representing the received frame of 
5 analog image data is stored in memory for analysis and manipulation by the software program. 

The computer receives a data record including text data and text-controlling 
parameters from text input 22. As described below the present software program manipulates 
both the digitized image of the received frame of analog image data and the text data from text 
input 22 and combines the video and text data onto the projection screen 12 in a generally 
^JPO smooth and continuous manner. 

?~j Text input 22 may include any of a variety of conventional input devices, such as 

§1 a hard drive memory, disc memory, compact disc, or a conventional Internet connection (such as 
« through a connected modem), or through the use of a connected keyboard. Alternately, the text 

H : may be supplied to the computer directly from a word processing program, such as Corel's 
;f 5 WordPerfect®, or Microsoft Word® as a text file or by copying selected text from an open text 
5 file directly into the program. Text data is preferably inputted into the memory 21 of computer 
20 for later retrieval by the present program and displayed in accordance with the invention, or 
alternately, the text is inputted in realtime for immediate projection onto screen 12. 

An example of text data 30 for use with the present invention is shown in Fig. 4 
20 (shown displayed on a computer monitor 32): "When it rains, it pours. When it stops, its wet." 
This exemplary text data 30 is inputted into the computer 20 by any conventional means and is 
parsed into appropriately-sized lines 31 (depending on the size of the projection screen 12). If 
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the text data 30 exceeds a certain number of characters 36, the program will parse the text data 30 
into separate lines of text 3 1 so that each line may fit onto the screen 12 (the wider the screen 12, 
the longer each line of text 31). The program will form a line data record to keep track of which 
lines of text 31 are on the screen 12, are fading from the screen 12, and should be introduced at 
5 the top of the screen. Each line of text 3 1 located on the line data record will be introduced in a 
prearranged order and every line of text 3 1 which is being projected on the screen 12 will be 
electronically marked or flagged in the line data record. In this manner, as one line of text 3 1 
fades from the screen 12, its electronic mark is removed from the line data record and a 
subsequent line of text 3 1 will be introduced onto the screen 12. 

Each line located in the line data record, is also stored in a character data record 
u 34, shown in Fig. 5. According to one aspect of the invention, the character data record 34 may 
Q further include various parameters which will control how each character 36 of each line of text 
« 3 1 moves and appears within the virtual environment, as viewed on screen 12. The parameters 

may vary depending on the desired effect of the characters 36 of each line of text 3 1 within the 
yfc virtual environment and depending on the complexity and speed of the interaction system. 

The controlling parameters may include a fall rate value which controls the speed 
each character 36 of each line of text 31 appears to fall on the screen 12. The fall rate value may 
be defined in a variety of ways, such as the number of pixels traveled in one second or between 
captured frames of video. 
20 Another character parameter may be a fall delay value which determines the 

length of time each character 36 will remain at the top of the screen 12 prior to falling. 
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A color code may be assigned to each character 36 or all of the characters within 

each line of text 3 1 . 

Another parameter for each character 36 is a horizontal hold condition which, 
when set, restricts each character to vertical movement when projected within the virtual 
5 environment. 

A fade parameter controls whether a character 36 fades when reaching the bottom 
of the screen 12 within the virtual environment. 

An edge-detect parameter determines whether a character will recognize the 
virtual image of a user within the virtual environment (i.e., determines if a virtual user interacts 
J|p with the falling characters). 

r~f The distance the characters 36 "fall" along the screen 12 within the virtual 

o jr 

U environment may also be controlled using a maximum fall distance parameter. 

« The above-listed character parameters are provided to illustrate that each character 

36 may be controlled in a variety of ways independently of the other characters 36 within a line 

35 of text 3 1 . Although a variety of parameters are possible, as exemplified above, it is preferred 
that most, if not all, include default values which simplify the setup and use of the present 
program. It is preferred that an operator merely be required to input the text data 30 into 
computer 20 without further setup. 

In operation, referring to Fig. 6, the present program first receives input text data 
20 30 at step 100. The text data 30 is first divided up into lines of text 3 1 at step 1 10. The number 
of characters 36 in each line of text 3 1 depends, in part, on the dimensions of the viewing screen 
12. The lines of text 31 may be divided into any number of characters, following, for example, 
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the length of each line of a poem. A fall rate value (and perhaps other variables) is assigned to 
each character 36 of each line of text 3 1 at step 120. As described above, it is preferred that all 
the characters 36 of any given line of text 3 1 have the same fall rate. At step 130, a frame of 
video from the camera is received and digitized by the video capture card 24, generating a bitmap 
5 wherein an RGB color value is assigned to each pixel defined in the captured image frame. Once 
the program starts capturing frames of video data, the program enters into a callback mode 
whereby every time a new frame of video data is received, select pixels of the bitmap are 
analyzed, as described below. Since only select pixels are analyzed by the present program, the 
speed of the program is much faster than prior art VR systems which rely on analyzing each pixel 
J¥0 in the bitmap for each frame of image data received. 

sj* At step 140, the program determines which lines of text 3 1 are to be displayed on 

fi the screen 12. It is preferred that three lines of text 3 1 be displayed on the screen 12 at any given 
35 time. The number of lines of text 3 1 may vary, of course, depending on the particular 

J: application, the size of the viewing screen 12, and the desired effect. At step 150, in Fig. 6, the 

3jj5 program determines the destination for a selected character 36 among the characters of the "to be 
displayed" lines of text 31, according to the predetermined fall rate of the selected character 36. 
At step 160, the color value of the "destination pixels" or those pixels which are located at the 
destination of the selected character 36 at step 150. The program, at step 170, then compares the 
read color value of the destination pixels with a threshold color (or contrast) value. At step 180, 
20 if it is determined at the comparison step 80 that the destination pixel color value is greater than 
the threshold color value, then the destination of the selected character 36 is raised by a 
predetermined number of pixels, preferably a single pixel, at step 190. In this instance, the pixel 
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color value at the newly determined destination of the selected character 36 is again read at step 
160, its value compared again with the threshold value at step 170 and a decision is again made 
at step 1 80 until the read pixel value of the destination pixel is less than or equal to the threshold 
color value. 

At this point, the determined destination of the selected character 36 is stored in 
memory at step 200. At step 210, it is determined if any more characters 36 remain within the 
"to be displayed" lines of text 3 1 . If there are more characters 36 remaining within the "to be 
displayed" lines of text 31, another character 36 is selected at step 220 and again the destination 
of the newly selected character 36 is determined at step 150 and the above-described process is 
repeated until the destination of each character 36 of the "to be displayed" lines of text 3 1 have 
been determined. When this occurs, the captured video image frame together with each character 
36 of the "to be displayed" lines of text 31 are displayed onto screen 12 (at step 230), with each 
character 36 located at each respective destination, as determined by the program and as stored at 
step 200. 

The program then returns to step 130 to acquire another image frame of video 
from camera 1 8 and repeats the above described steps, as shown in the flow diagram of Fig. 6. 

The end result is that the individual characters 36 of their respective "to be 
displayed" lines of text 3 1 will appear to fall (at their prescribed fall rate) from the top of the 
projection screen 12, as illustrated in Figs. 7 and 8. In Fig. 7, an exemplary projected image on 
screen 12 is shown to have a shoulder 240, and an arm 242 of a virtual user 244. The actual user 
(not shown) and the virtual user 244 are holding an umbrella 246 which appears on the screen 12 
in this example, as a virtual umbrella 246. As is shown, the program has displayed the video 
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image of the virtual user (shown in part) 244 holding the virtual umbrella 246 and the inputted 
lines of text 3 1 : "When it rains, it pours". In this exemplary scene, a new line of text 3 1 : "When 
it rains, it pours" has just been introduced at the top portion of the screen 12, generally indicated 
by arrow 248 of Fig. 7. The characters "h" and "e" of the word "When" have already started to 
fall at their quicker fall rates. Various characters 36 from previously introduced lines of text 31 
(in this example, all the lines of text 31 shown on the screen 12 are the same: "When it rains, it 
pours") have been drifting downwardly from the top portion 248 of the screen 12 to towards the 
bottom portion of the screen, generally indicated by arrow 250 in Fig. 7. 

Following the steps described above, as the computer 20 determines the 
destination of each character 36, it quickly reads the pixel color values at that destination of the 
captured image and determines whether a virtual obstacle is already located there. If no obstacle 
is there (i.e., the read pixels are less than or equal to a threshold value), then the particular 
character 36 appears on the screen 12 at the destination and, as subsequent images are displayed 
on screen 12, the particular character 36 will appear to slowly fall, like snow. If however, a 
virtual obstacle is determined to be located at the destination of a particular character 36, then a 
new destination located higher on the screen 12 is selected so that the character 36 appears to 
interact with the virtual obstacle located in the virtual environment. This interaction is illustrated 
in Fig. 8, wherein an enlarged partial section of the umbrella 246 is shown "capturing" the 
characters "r", "a", and "i" of the word "rains". These characters 36 will remain at the first 
"light" pixel (below the pixel color threshold value) located above the relatively dark pixels 
which make up the umbrella 246 so that the characters 36 appear to have fallen onto and are 
supported by the umbrella 246. If the user raises the umbrella 246, the program will quickly 
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detect this upward movement because the "lighter" pixel between the character 36 and the 
umbrella 246 will become a dark value causing the computer to respond by raising the affected 
characters 36 upwardly with the umbrella movement, so that the characters 36 appear to have 
been lifted by the umbrella 246. The characters 36 can move upwardly very quickly, but will 
5 preferably always fall at the predetermined fall rate, as discussed above. 

As subsequent video images are sampled, each character 36 of a line of text 31 
appear to gradually descend on the screen 12 until the character 36 confronts a dark pixel, 
signifying a virtual obstacle within the virtual environment. Although the present program may 
be setup so that the characters 36 fall at any fall rate, it is preferred that the characters 36 fall in a 
ctO manner which is somewhat familiar and pleasing to the user, similar, for example, to the graceful 
i ■ movements of an airborne leaf slowing falling from a tree. A user can "catch" the falling leaf 

U and quickly push it back into the wind currents, but the leaf will always fall back down at its 

" leisurely pace. By "training" the introduced characters 36 to respond in this manner within the 

if, virtual environment, the user is likely to remain comfortable and will be encouraged to further 

iy 

f 4 5 interact and explore within the virtual environment. 

The text data 30 may be the lines of a poem, song, or even the tag-line of an 
advertisement of a particular product and may be projected in any size, shape or font (system 
fonts are preferably used so that the processing is faster). As the user interacts with the 
introduced lines of text 3 1, he or she must physically manipulate the falling characters 36 and 
20 perhaps capture several of them to read them and understand the actual text, but the fun of 

combining such a physical activity with reading only encourages the user to succeed in reading 
the textual information. Whatever message the textual information is to convey is more likely to 

Page 18 of 30 



ii 



reach the viewer or user when the user gets to participate with the message in an interactive 
game-like environment. 

The textual information introduced into the present program may be continuously 
changed, such as in the case of weather reports, or news stories. Alternately, the textual 
information may be altered in realtime, perhaps in response to the particular actions of the user. 
In this instance, a user may be asked questions in a text format with the question appearing on the 
screen 12 together with the image of the user. When the user reads the question on the screen 12, 
perhaps after capturing the characters 36, he or she can audibly answer the question. The 
response to the question can be heard by the operator of the system, who may then ask another 
question (again in text format) which relates or corresponds to the user's response to the previous 
question. This type of interaction between the text-input operator and the user may further 
entertain and teach the user and any viewer of the system. 

According to another aspect of the invention, text data may be introduced one line 
at a time allowing a first line of text 3 1 to be introduced at the top of the of the viewing screen 12 
so that each character 36 begins to fall (as viewed on the viewing screen 12), either at similar fall 
rates or at different ones. When a predetermined number of falling characters 36 (e.g., at least 
half of them) travels a predetermined distance from the top of the viewing screen 12, a second 
line (or select characters of the second line) of text 31 will appear at the top of the screen 12 . 
The characters 36 of the newly introduced second line of text 3 1 will begin to fall along the 
viewing screen 12 in a similar manner to those of the first line of text. Third and other lines of 
text 3 1 may be displayed in a similar and repeating manner. 
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Another aspect of the invention causes inputted lines of text 3 1 to fade (at a 
prescribed fade rate) when a predetermined number of characters (e.g., at least half of them) 
reach a predetermined position on the viewing screen (e.g., enter the lower half of the screen). 
In a preferred embodiment, at any given moment, three lines of text 3 1 are displayed on the 
viewing screen 12, one line of text 31 being introduced at the top of the screen, a second one 
falling in the middle of the screen (an "interactive region"), and a third line of text 31 fading out 
at the lower portion of the screen. Of course, some of the characters 36 (or letters) of earlier 
introduced lines of text 31 may remain on the screen, "captured" by virtual obstacles, such as the 
above-described umbrella 246 of Fig. 7. If one particular character 36 from a first line of text 3 1 
remains held in vertical space by a virtual obstacle (e.g., umbrella 246), the same character 36 
from subsequent lines of text 3 1 will eventually overlay the first "held" character 36 so that 
similar characters 36 held by virtual obstacles will not accumulate in number over time. 
However, it is also contemplated that as characters 36 descend onto each other on the screen 12, 
the program can be altered so that any higher character 36 will consider a lower "held" character 
to be part of the virtual obstacle so that the characters will pile up (stack on top of each other), 
until the obstacle is cleared (i.e., the user moves). 

According to another aspect of the invention, each line of text 3 1 is displayed as a 
separate color. In this arrangement, viewers may easily discern the characters from different 
lines of text 3 1 so that the lines of text 3 1 to be read relatively easily even as the individual 
characters 36 fall towards and rest on various darker elements located within the virtual 
environment, such as the arm of a virtual user and perhaps a "virtual umbrella image" carried by 
the virtual user. 
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According to another aspect of the invention each character 36 of inputted text 
data 30 is horizontally-held, so that each character 36 is restricted to vertically movement as it 
descends on the viewing screen 12 at its prescribed fall rate, or ascends with upward movement 
of a virtual obstacle. 

5 The techniques described herein for integrating a user's image and inputted textual 

data within a digitally represented environment can be applied to numerous applications 
including fun and educational communication applications for use in schools to help kids learn to 
read or just have fun. In this application, Kids could be asked, for example, to interact with a 
falling sentence by "capturing" a verb located within the falling sentence. 
flO The present system can also be installed at a store front, in which case a display 

/; monitor may be positioned within the store so that it may be viewed by passing pedestrians. In 
2 this application, the appearance of a pedestrian on the monitor will invariably attract the attention 
- of the pedestrian, who will stop and interact with the textual message falling all around his or her 

•T Hi 
I* ">><■ ;< 

H image being displayed on the screen. The message, in this example, would help lure customers 
;fj 5 inside the store to purchase products. 

r "~ Other useful applications for this technology include performance art, either used 

in connection with an artist or an audience member. The present invention can be setup within 

an art gallery, exhibit, or museum as an interesting and alluring way to convey any textual 

information either as a piece of art work, or as a source of information, such as an interactive 

20 information kiosk. 

According to another aspect of the invention, the present software is connected to 

the Internet through appropriate modems, Tl, or DSL communication lines for the purpose of 
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allowing Internet users to introduce text data 30 to a particular interactive display shown over the 
Internet. Also, the software program may be used with a local PC whereby the computer's 
monitor becomes the viewing screen, and a local "webcam" type camera (or any other 
appropriate camera) may be used to record the user, who in this case, would be positioned in 
front of the computer. If a CRT or LCD type monitor is used as the screen 12, the camera may 
be positioned on top of (or adjacent to) the monitor.. 

Referring to Fig. 9, an enlarged partial screen 12 showing the central opening 14 
is shown, according to another embodiment of the invention. Applicant has discovered that light 
from projector 16 directed to the optical input lens of camera 18 causes the images captured by 
the camera to appear "washed out" because the camera automatically compensates for correct 
exposure. To overcome this problem, the present program generates a shadow zone 13 of dark 
pixels which are projected onto the viewing screen 12. The shadow zone 13 is aligned with the 
optical input of the digital camera so that the camera will be protected from the relatively bright 
light of the projector. The computer-generated shadow zone 13 helps prevent the images 
recorded by the camera from becoming "washed out" by the light from the projector shining on 
the camera's lens. The shadow zone 13 is shown as a darkened square, however, the shadow 
zone 13 may take on any useful shape to cover opening 14, and also may be any shade of gray. 
The program includes means to project the shadow zone 13 when each image and lines of text 31 
are projected onto screen 12. Also, the projected position of the shadow zone 13 on screen 12 
may be easily moved by the program, using an appropriate input device, such as a mouse or a 
keyboard (not shown). 
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The present software program (see appendix for actual source code) is written in 
C++ programming language using standard Microsoft® and Video-for- Windows® Libraries. 

Although the above detailed description of the invention assumes the presence of 
a single user, the invention is capable of simultaneously supporting multiple users. These users 
5 can occupy the same space in front of a single camera 1 8, or images of each user can be inputted 
to a single computer 20 using separate cameras at physically disparate locations. In the latter 
case, remote image data reaches the invention by means of suitable network connections, and the 
invention sends picture signals to the user's remote monitor over the same facility. In this way, 
all users interact in the same virtual world. 
flO The terms and expressions employed herein are used as terms of description and 

":r, it 

7 s } not of limitation, and there is no intention, in the use of such terms and expressions, of excluding 

H' any equivalents of the features shown and described or portions thereof, but it is recognized that 

i: various modifications are possible within the scope of the invention claimed. 
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